Active-by-active programmable device

ABSTRACT

An example integrated circuit (IC) system includes a package substrate having a programmable integrated circuit (IC) and a companion IC mounted thereon, the programmable IC including a programmable fabric and the companion IC including application circuitry. The IC system further includes a system-in-package (SiP) bridge including a first SiP IO circuit disposed in the programmable IC, a second SiP IO circuit disposed in the companion IC, and conductive interconnect on the package substrate electrically coupling the first SiP IO circuit and the second SiP IO circuit. The IC System further includes first aggregation and first dispersal circuits in the programmable IC coupled between the programmable fabric and the first SiP IO circuit. The IC system further includes second aggregation and second dispersal circuits in the companion IC coupled between the application circuitry and the second SiP IO circuit.

This application is an application for reissue of U.S. Pat. No.10,002,100, which is incorporated herein by reference in its entirety.

TECHNICAL FIELD

Examples of the present disclosure generally relate to electroniccircuits and, in particular, to an active-by-active programmable device.

BACKGROUND

Modern programmable devices, such as field programmable gate arrays(FPGAs), are growing in size and becoming more heterogeneous. Their costis also rapidly increasing due to both more expensive process technologyand increasing overhead of programmability for a majority ofapplications that do not require all of the heterogeneous circuitblocks. Many of these large circuit blocks, such as general purposeinput/output (IO) or multi-gigabit serial transceivers (MGTs), do notrequire the benefits of new process technology. Thus, the traditionalmonolithic architectures no longer meet the cost requirements of themarket, leading to the development of system-in-package (SiP) devices.The majority of SiP solutions, however, rely on advanced packagingtechniques, such as the use of expensive interposers or complexthree-dimensional die stacking. As such, the added cost of these SiPsolutions limits the benefits to high-end or niche applications with lowproduction volume.

SUMMARY

Techniques for providing an active-by-active programmable device aredescribed. In an example, an integrated circuit (IC) system includes apackage substrate having a programmable integrated circuit (IC) die anda companion IC die mounted thereon, the programmable IC die including aprogrammable fabric and the companion IC die including applicationcircuitry. The IC system further includes a system-in-package (SiP)bridge including a first SiP IO circuit disposed in the programmable ICdie, a second SiP IO circuit disposed in the companion IC die, andconductive interconnect on the package substrate electrically couplingthe first SiP IO circuit and the second SiP IO circuit. The IC Systemfurther includes first aggregation and first dispersal circuits in theprogrammable IC die coupled between the programmable fabric and thefirst SiP IO circuit. The IC system further includes second aggregationand second dispersal circuits in the companion IC die coupled betweenthe application IO and the second SiP IO circuit.

In another example, a programmable integrated circuit (IC) includes asystem-in-package (SiP) input/output (IO) circuit coupled to a companionIC through external conductive interconnect; a programmable fabricwithout at least a portion of application circuitry; and aggregation anddispersal circuits coupled between the programmable fabric and the SiPIO circuit.

In another example, a method of transmitting data from a programmableintegrated circuit (IC) in an IC system includes coupling the data to afirst system-in-package (SiP) IO circuit through a plurality of channelsof an aggregation circuit in the programmable IC. The method furtherincludes transmitting the data from the plurality of channels over asmaller number of physical channels between the programmable IC and acompanion IC. The method further includes receiving the data from theplurality of physical channels at a second SiP IO circuit in thecompanion IC. The method further includes coupling the data from thesecond SiP IO circuit to application circuitry in the companion ICthrough a plurality of channels of a dispersal circuit in the companionIC. The method further includes transmitting the data from theapplication IO circuits.

These and other aspects may be understood with reference to thefollowing detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features can be understoodin detail, a more particular description, briefly summarized above, maybe had by reference to example implementations, some of which areillustrated in the appended drawings. It is to be noted, however, thatthe appended drawings illustrate only typical example implementationsand are therefore not to be considered limiting of its scope.

FIG. 1 is a block diagram depicting an integrated circuit (IC) systemaccording to an example.

FIG. 2 is a cross-section of a multi-chip module (MCM) according to anexample.

FIG. 3 illustrates an example field programmable gate array (FPGA)architecture.

FIG. 4 is a block diagram depicting an IC system according to anotherexample.

FIG. 5 is a block diagram depicting a more detailed view of the IC dieof FIG. 4 according to an example.

FIG. 6 is a block diagram depicting system-in-package (SiP) stacksaccording to an example.

FIG. 7 is a block diagram depicting a data link layer in more detailaccording to an example.

FIG. 8 is a flow diagram depicting a method of arbitration within thedata link layer of an SiP bridge according to an example.

FIG. 9 is a flow diagram depicting a method of arbitration within thedata link layer of a SiP bridge according to another example.

FIG. 10 is a flow diagram depicting a method of transmitting data from aprogrammable IC in an IC system according to an example.

FIG. 11 is a flow diagram depicting a method of receiving data to aprogrammable IC in an IC system according to an example.

FIG. 12 is a block diagram depicting physical layer logic according toan example implementation.

FIG. 13 is a block diagram depicting an integration of system-levelinterconnect with a horizontal edge of programmable fabric according toan example.

FIG. 14 is a block diagram depicting an integration of system-levelinterconnect with a vertical edge of programmable fabric according to anexample.

FIG. 15 is a block diagram depicting a ring sub-node of system-levelinterconnect according to an example.

FIGS. 16A through 16F show example system-level interconnect ring (SIR)implementations.

FIG. 17 is a flow diagram depicting a method of transmitting data from aprogrammable IC in an IC system according to an example.

To facilitate understanding, identical reference numerals have beenused, where possible, to designate identical elements that are common tothe figures. It is contemplated that elements of one example may bebeneficially incorporated in other examples.

DETAILED DESCRIPTION

Various features are described hereinafter with reference to thefigures. It should be noted that the figures may or may not be drawn toscale and that the elements of similar structures or functions arerepresented by like reference numerals throughout the figures. It shouldbe noted that the figures are only intended to facilitate thedescription of the features. They are not intended as an exhaustivedescription of the claimed invention or as a limitation on the scope ofthe claimed invention. In addition, an illustrated example need not haveall the aspects or advantages shown. An aspect or an advantage describedin conjunction with a particular example is not necessarily limited tothat example and can be practiced in any other examples even if not soillustrated, or if not so explicitly described.

An active-by-active programmable device is described. In an example, asystem-in-package (SiP)-based programmable device employs a multi-chipmodule (MCM) package. The MCM package includes a programmable integratedcircuit (IC), such as a field programmable gate array (FPGA), and one ormore companion integrated circuit (IC) devices disposed on a packagesubstrate. The programmable IC and the companion ICs are disposedside-by-side on the package substrate (e.g., active-by-active). Theconnection between the programmable IC and each companion IC device isimplemented using a high-bandwidth SiP bridge. The SiP bridge can beimplemented using a low number of wires, allowing use of the MCM packagerather than an expensive interposer. Data to be sent from one device toanother is aggregated into a collective bandwidth and delivered over theSiP bridge. Aggregate data received on the SiP bridge is delivered tothe destination through a systematic disperse mechanism. In examplesdescribed herein, the SiP bridge is implemented using a protocol stackcomprising at least physical and data link layers. Higher layers canalso be employed, such as a transport layer. The physical layer can beany ultra-short reach (USR) serializer/deserializer (SerDes) technologythat meets certain requirements detailed herein. As described furtherherein, the data link layer is configured to time-multiplex theaggregated data across the available physical channels in a manner thatavoids congestion at the destination. Each device can include asystem-level interconnect to facilitate aggregation and dispersal ofdata between application circuits and SiP bridge(s).

In examples, a companion IC die includes some or all of the applicationinput/output (IO) circuits of the SiP device. Thus, the programmable ICcan be constructed without such application IO circuits. Removal ofapplication IO circuits from the programmable IC frees die area for useby other circuits. Also, the programmable IC can be manufactured usingnewer process technology, taking advantage of newer process technologyfeatures, while the companion IC having the application IO circuits canbe manufactured using older, less expensive process technology. As aresult, the overall solution will deliver the same functionality at alower cost. The programmable IC and companion IC device are connected bya SiP bridge. Each of the programmable IC and the companion IC deviceinclude a SiP IO circuit defining an endpoint of the SiP bridge. Each ofthe programmable IC and the companion IC can include system-levelinterconnect to provide aggregation and dispersal of data betweenapplication circuits and the SiP IO circuits. One example of asystem-level interconnect for use in the programmable IC is asystem-level interconnect ring (SIR). As described further herein, theSIR allows for minimal augmentation of the programmable fabric and thedesign tools used to implement circuits for the programmable IC. Theseand other aspects can be understood with reference to the followingfigures.

FIG. 1 is a block diagram depicting an integrated circuit (IC) system100 according to an example. The IC system 100 includes an IC die 101coupled to an IC die 103. In an example, the IC die 101 and the IC die103 are part of a multi-chip module (MCM) forming a System-in-Package(SiP). The IC die 101 includes application circuits 105, aggregationcircuits 110, dispersal circuits 112, and a SiP input/output circuit(SiP IO 140). In an example, the application circuits 105 includevarious components, such as one or more processing elements (PE(s) 102),memory (mem 104), support circuits (SC 106), and input/output circuits(IO 108). In other examples, one or more of the components 102-108 maybe omitted from the application circuits 105. Each of the components 102through 108 of the application circuits 105 can be coupled to both theaggregation circuits 110 and the dispersal circuits 112. Alternatively,one or more of the components 102-108 may be coupled only to one of theaggregation circuits 110 or the dispersal circuits 112. The aggregationcircuits 110 and the dispersal circuits 112 are each coupled to aninternal interface of the SiP IO 140.

In an example, SiP IO 140 implements multiplexed IO logic havingtransport logic 114, a data link logic 116, and physical logic 118. Inthe example shown, the multiplexed IO logic is implemented entirelywithin SiP IO 140. In other examples, a given layer of the multiplexedIO logic or a portion thereof can be implemented in the applicationcircuits 105. In an example, the transport logic 114 is implemented inthe application circuits 105 and the data link logic 116 and thephysical logic 118 are implemented in the SiP IO 140. In an example, anarbitration portion of the data link logic 116 is implemented in theapplication circuits 105 and multiplexing logic for the data link logic116 is implemented in the SiP IO 140.

The IC die 103 can include similar circuitry as the IC die 101. In theexample shown in FIG. 1, the IC die 103 includes application circuits107, aggregation circuits 126, dispersal circuits 128, and a SiP IO 142.In an example, the application circuits 107 include various components,such as PE(s) 120, memory (mem 132), support circuits (SC 134), andinput/output circuits (IO 136). In other examples, one or more of thecomponents 130-136 may be omitted from the application circuits 107.Each of the components 130 through 136 of the application circuits 107can be coupled to both the aggregation circuits 126 and the dispersalcircuits 128. Alternatively, one or more of the components 130-136 maybe coupled only to one of the aggregation circuits 126 or the dispersalcircuits 128. The aggregation circuits 126 and the dispersal circuits128 are each coupled to an internal interface of the SiP IO 142. Similarto the SiP IO 140, the SiP IO 142 implements multiplexed IO logic havingtransport logic 124, a data link logic 122, and physical logic 120. Inthe example shown, the multiplexed IO logic is implemented entirelywithin SiP IO 142. In other examples, as described above with respect tothe IC die 101, a given logic layer of the multiplexed IO logic or aportion thereof can be implemented in the application circuits 107.

The IC die 101 and the IC die 103 are coupled by a SiP bridge 144. TheSiP bridge 144 includes the SiP IO 140, SiP IO 142, and signal paths138. An external interface of the SiP IO 140 is coupled to an externalinterface of the SiP IO 142 by the signal paths 138. The physical logic118/120 implement a physical layer of the SiP bridge 144 and support aplurality of physical channels. The data link logic 116/122 implement adata link layer of the SiP bridge 144 and support a plurality ofchannels on each side of the SiP bridge 144 (referred to as aggregationchannels and dispersal channels). The transport logic 114/124 implementa transport layer of the SiP bridge 144. Various transport layers can beemployed, including connection-less or connection-based transportlayers. The transport layer can provide for packetization,de-packetization, error correction, packet ordering, and the like knownin the art.

The application circuits 105 can include a number of outputs coupled tochannels of the aggregation circuits 110. The aggregation circuits 110selectively couple outputs of the application circuits 105 amonginternal inputs of the SiP IO 140 (referred to as source ports). Forexample, the SiP IO 140 can include M source ports coupled to Maggregation channels of the aggregation circuits 110, where M is apositive integer. The aggregation circuits 110 can selectively coupleoutputs of the application circuits 105 to the M source ports of the SiPIO 140 through the M aggregation channels. The SiP IO 140 can include Kexternal outputs driving K physical channels implemented over the signalpaths 138, where K is a positive integer. In an example, K is less thanM and the SiP IO 140 multiplexes the M source ports among the K externaloutputs. The SiP IO 142 can include K external inputs receiving from theK physical channels. The SiP IO 142 de-multiplexes the K external inputsamong N internal outputs, where N is a positive integer (referred to asdestination ports). In an example, N is greater than K. The dispersalcircuits 128 selectively couple the N destination ports of the SiP IO142 among inputs of the application circuits 107 through N dispersalchannels.

Likewise, the application circuits 107 can include a number of outputscoupled to aggregation channels of the aggregation circuits 126. Theaggregation circuits 126 selectively couple outputs of the applicationcircuits 107 among source ports of the SiP IO 142. For example, the SiPIO 142 can include N′ source ports coupled to N′ aggregation channels ofthe aggregation circuits 126, where N′ is a positive integer. Theaggregation circuits 126 can selectively couple outputs of theapplication circuits 107 to the N′ source ports of the SiP IO 142through the N′ aggregation channels. The SiP IO 142 can include K′external outputs driving K′ physical channels implemented over thesignal paths 138, where K′ is a positive integer. In an example, K′ isless than N′ and the SiP IO 142 multiplexes the N′ source ports amongthe K′ external outputs. The SiP IO 140 can include K′ external inputsreceiving from the K′ physical channels. The SiP IO 140 de-multiplexesthe K′ external inputs among M′ destination ports, where M′ is apositive integer. In an example, M′ is greater than K′. The dispersalcircuits 112 selectively couple the M′ destination ports of the SiP IO140 among M′ inputs of the application circuits 105 through M′ dispersalchannels.

In some examples, K=K′ such that the the SiP bridge 144 supports anequal number of physical channels in each direction between the IC die101 and the IC die 103. To transmit data from the IC die 101, theaggregation circuits 110 aggregate output of the application circuits105 into M aggregation channels and the SiP IO 140 multiplexes the Maggregation channels across K physical channels. To receive the data atthe IC die 103, the SiP IO 142 de-multiplexes K physical channels into Nchannels and the dispersal circuits 128 disperse the N dispersalchannels to inputs of the application circuits 107. Likewise, totransmit data from the IC die 103, the aggregation circuits 126aggregate output of the application circuits 107 into N′ aggregationchannels and the SiP IO 142 multiplexes the N′ aggregation channelsacross K′ physical channels. To receive the data at the IC die 101, theSiP IO 140 de-multiplexes K′ channels received from the K′ physicalchannels into M′ dispersal channels and the dispersal circuits 112disperse the M′ dispersal channels to inputs of the application circuits105. In some examples, M=M′ and N=N′ such that aggregation circuits 110and the dispersal circuits 112 provide a total of 2*M channels to theapplication circuits 105 and the aggregation circuits 126 and thedispersal circuits 128 provide a total of 2*N channels to theapplication circuits 107. In some examples, M=M′=N=N′.

In an example, the application circuits 105 and the application circuits107 exchange packetized data over the SiP bridge 144. The transportlogic 114 forms the output of the application circuits 105 into packetseach including w bits, where w is a positive integer. The transportlogic 114 also de-packetizes data received from the dispersal circuits112. The transport logic 124 can function similarly to the transportlogic 114.

The data link logic 116 organizes the output of the aggregation circuits110 into a plurality of aggregation channels (e.g., M aggregationchannels), where each aggregation channel is w-bits wide for providing aw-bit packet. That is, the aggregation circuits 110 include amulti-channel output. Likewise, the data link logic 116 divides theinputs of the dispersal circuits 112 into a plurality of dispersalchannels (e.g., M′ dispersal channels), where each dispersal channel isw-bits wide for receiving a w-bit packet. That is, the dispersalcircuits 112 include a multi-channel input. The data link logic 116maintains a transmit queue for each aggregation channel. The data linklogic 116 arbitrates among the transmit queues to select packets to betransmitted over the available physical channels by the physical logic118. The data link logic 116 also de-multiplexes physical channels intothe available dispersal channels of the dispersal circuits 112. The datalink logic 122 can function similarly to the data link logic 116.

The physical logic 118 can serialize packets in the available outputphysical channels for transmission as signals across the signal paths138. The physical logic 118 can also de-serialize signals received fromthe signal paths 138 into the available input physical channels. Thephysical logic 120 can function similarly to the physical logic 118.

In an example, the application circuits 105 can be coupled to theapplication circuits 107 directly over one or more direct signal paths146. That is, there can be some communication between the IC die 101 andthe IC die 103 that does not traverse the SiP bridge 144. Further, theIC die 101 can include an external interface 147 that can be used tosend and receive signals to other external circuits, as well as toreceive power and ground. Likewise, the IC die 103 can include anexternal interface 148 that can be used to send and receive signals toother external circuits, as well as to receive power and ground.

In the example shown in FIG. 1, the IC die 101 includes SiP IO for oneSiP bridge to one other IC die. In other examples, an IC die can includemultiple SiP IOs for multiple SiP bridges to multiple IC die. In suchembodiments, the IC die can include multiple instances of aggregationand dispersal circuits to couple the application circuits therein to themultiple SiP bridges.

FIG. 2 is a cross-section of a MCM 200 according to an example. The MCM200 can implement the IC system 100 described above. The MCM 200includes the IC die 101 and the IC die 103 mounted on a packagesubstrate 202. The IC die 101 is electrically and mechanically coupledto the package substrate 202 by contacts 214. The IC die 103 iselectrically and mechanically coupled to the package substrate 202 bycontacts 216. The contacts 214 and the contacts 216 can be solder balls,solder bumps, or the like. The package substrate 202 includes conductiveinterconnect 208 disposed at or near the surface to which the IC die 101and the IC die 103 are mounted. The package substrate 202 includesconductive interconnect 206 disposed at or near the opposite surface.The conductive interconnect 208 and the conductive interconnect 206 eachincludes one or more metal layers having a specific pattern. Some of thecontacts 214 of the IC die 101 are electrically coupled to theconductive interconnect 208. Some of the contacts 216 of the IC die 103are electrically coupled to the conductive interconnect 208. Others ofthe contacts 214 are electrically coupled to the conductive interconnect206 through vias 218. Others of the contacts 216 are electricallycoupled to the conductive interconnect 206 through vias 220. Theconductive interconnect 206 is electrically coupled to contacts 204. Thecontacts 204 can be solder balls, solder bumps, or the lie.

The contacts 204 comprise the external pins of the MCM 200. The MCM 200can be mounted to a circuit board. Conductors on the circuit board canbe electrically coupled to the IC die 101 and the IC die 103 through thecontacts 204 of the package substrate 202. Within the MCM 200, the ICdie 101 is electrically coupled to the IC die 103 through the conductiveinterconnect 208. To implement the IC system 100 described above, theconductive interconnect 208 is patterned to form the signal paths 138between the IC die 101 and the IC die 103. In an example, the conductiveinterconnect 208 is also patterned to form the dedicated signal path(s)146 (if present). The external interface of the SiP IO 140 is coupledthe signal paths 138 formed in the conductive interconnect 208 throughsome of the contacts 214. Likewise, the external interface of the SiP IO142 is coupled to the signal paths 138 formed in the conductiveinterconnect 208 through some of the contacts 216.

In an example, at least one of the IC die 101 and the IC die 103comprises a programmable IC, such as a field programmable gate array(FPGA) or the like. FIG. 3 illustrates an FPGA architecture 300 thatincludes a large number of different programmable tiles includingmulti-gigabit transceivers (“MGTs”) 301, configurable logic blocks(“CLBs”) 302, random access memory blocks (“BRAMs”) 303, input/outputblocks (“IOBs”) 304, configuration and clocking logic (“CONFIG/CLOCKS”)305, digital signal processing blocks (“DSPs”) 306, specializedinput/output blocks (“I/O”) 307 (e.g., configuration ports and clockports), and other programmable logic 308 such as digital clock managers,analog-to-digital converters, system monitoring logic, and so forth.Some FPGAs also include dedicated processor blocks (“PROC”) 310.

In some FPGAs, each programmable tile can include at least oneprogrammable interconnect element (“INT”) 311 having connections toinput and output terminals 320 of a programmable logic element withinthe same tile, as shown by examples included at the top of FIG. 3. Eachprogrammable interconnect element 311 can also include connections tointerconnect segments 322 of adjacent programmable interconnectelement(s) in the same tile or other tile(s). Each programmableinterconnect element 311 can also include connections to interconnectsegments 324 of general routing resources between logic blocks (notshown). The general routing resources can include routing channelsbetween logic blocks (not shown) comprising tracks of interconnectsegments (e.g., interconnect segments 324) and switch blocks (not shown)for connecting interconnect segments. The interconnect segments of thegeneral routing resources (e.g., interconnect segments 324) can span oneor more logic blocks. The programmable interconnect elements 311 takentogether with the general routing resources implement a programmableinterconnect structure (“programmable interconnect”) for the illustratedFPGA.

In an example implementation, a CLB 302 can include a configurable logicelement (“CLE”) 312 that can be programmed to implement user logic plusa single programmable interconnect element (“INT”) 311. A BRAM 303 caninclude a BRAM logic element (“BRL”) 313 in addition to one or moreprogrammable interconnect elements. Typically, the number ofinterconnect elements included in a tile depends on the height of thetile. In the pictured example, a BRAM tile has the same height as fiveCLBs, but other numbers (e.g., four) can also be used. A DSP tile 306can include a DSP logic element (“DSPL”) 314 in addition to anappropriate number of programmable interconnect elements. An IOB 304 caninclude, for example, two instances of an input/output logic element(“IOL”) 315 in addition to one instance of the programmable interconnectelement 311. As will be clear to those of skill in the art, the actualI/O pads connected, for example, to the I/O logic element 315 typicallyare not confined to the area of the input/output logic element 315.

In the pictured example, a horizontal area near the center of the die(shown in FIG. 3) is used for configuration, clock, and other controllogic. Vertical columns 309 extending from this horizontal area orcolumn are used to distribute the clocks and configuration signalsacross the breadth of the FPGA.

Some FPGAs utilizing the architecture illustrated in FIG. 3 includeadditional logic blocks that disrupt the regular columnar structuremaking up a large part of the FPGA. The additional logic blocks can beprogrammable blocks and/or dedicated logic. For example, processor block310 spans several columns of CLBs and BRAMs. The processor block 310 canvarious components ranging from a single microprocessor to a completeprogrammable processing system of microprocessor(s), memory controllers,peripherals, and the like.

Note that FIG. 3 is intended to illustrate only an exemplary FPGAarchitecture. For example, the numbers of logic blocks in a row, therelative width of the rows, the number and order of rows, the types oflogic blocks included in the rows, the relative sizes of the logicblocks, and the interconnect/logic implementations included at the topof FIG. 3 are purely exemplary. For example, in an actual FPGA more thanone adjacent row of CLBs is typically included wherever the CLBs appear,to facilitate the efficient implementation of user logic, but the numberof adjacent CLB rows varies with the overall size of the FPGA.

In an example, the IC die 101 includes a programmable fabric having theFPGA architecture 300. Thus, the IC die 101 further includes the SiP IO140 and a system-level interconnect 350. The system-level interconnect350 includes the aggregation circuits 110 and the dispersal circuits112. In an example, the system-level interconnect 350 comprises asystem-level interconnect ring (SIR). Various examples of an SIR aredescribed below. In general, the system-level interconnect 350 providesan interface between the programmable fabric and the SiP IO 140. In anexample, the system-level interconnect 350 can be configured in similarfashion to the programmable fabric (e.g., via loading of a configurationbitstream). In other examples, the system-level interconnect 350 can bedynamically programmed during operation of the IC die 101.

In an example, the SiP IO 140 provides an IO interface that circuitsconfigured in the programmable fabric can use to communicate with the ICdie 103 over the SiP bridge 144. Circuits can use the application IO ofthe FPGA architecture 300, such as the IOBs 304, MGTs 301, and any otherIOs (e.g., memory IOs, custom IOs, etc), for additional IO to otherexternal circuits. In the example of FIG. 3, the FPGA architecture 300includes both the SiP IO 140 and application IO.

In other examples described in more detail below, at least a portion ofthe application IO can be disposed external to an FPGA, such as in acompanion IC of a MCM. For example, the IC die 101 having an FPGAarchitecture can be constructed with only the SiP IO 140. That is, theIOBs 304 and MGTs 301 are removed from the FPGA architecture 300. All ofthe application IO can be disposed in the IC die 103. The programmablefabric in the IC die 101 can access the application IO in the IC die 103using the SiP bridge 144. In some examples, the IC die 101 having anFPGA architecture can include some dedicated IO in addition to the SiPIO, such as configuration IO, JTAG IO, and the like. This dedicated IOis used for programming and/or testing the FPGA and not used asapplication IO for circuits configured in the programmable fabric.

FIG. 4 is a block diagram depicting an IC system 400 according toanother example. Elements in FIG. 4 that are the same or similar tothose of FIGS. 1 and 3 are designated with identical reference numerals.In the IC system 400, an IC die 101A is an FPGA die having aprogrammable fabric 404 coupled to the system-level interconnect 350. AnIC die 103A is an application specific integrated circuit (ASIC) diecomprising application IO 107A (also referred to as a “companion IC”).While the IC die 103A is described as having application IO 107A, ingeneral, the IC die 103A can include application circuitry, which mayinclude IO circuitry and/or other types of circuitry. The system-levelinterconnect 350 is coupled to an internal interface of SiP IO 140A. Thesystem-level interconnect 350 performs the functions of the aggregationcircuits 110 and the dispersal circuits 112 described above. The SiP IO140A is similar to the SiP IO 140 described above, but without thetransport logic 114 and a portion of the data link logic 116. Rather,the transport logic 114 is implemented in the programmable fabric 404.For example, the programmable fabric 404 can be configured withcircuits, such as PEs 102, that can implement the transport logic 114.The PEs 102 can be coupled to and use the memory 104 (e.g., BRAMs 303)and/or the support circuits 106 (e.g., DSPs 306). In another example,the PEs 102, the memory 104, and the support circuits 106 can alsoimplement arbitration logic 406 for the data link logic 116. The SiP IO140A includes multiplexing logic of the data link logic 116 and thephysical logic 118. In yet another example, the programmable fabric 404can implement both the arbitration logic 406 and the transport logic114.

In the example of FIG. 4, the system-level interconnect 350 is shown asa ring surrounding all four sides of the programmable fabric 404. Theprogrammable fabric 404 is coupled to the system-level interconnect 350on top, bottom, left, and right sides thereof. In other examplediscussed further below, the system-level interconnect 350 can haveother forms, such as different implementations of an SIR. In anotherexample, the system-level interconnect 350 can include a network-on-chip(NoC) or the like.

In an example, the programmable fabric 404 does not include anyapplication IO. Rather, application IO 107A in the IC die 103A includethe application IO, such as IOBs 304, MGTs 301, and any other IOs 402(e.g., memory IOs, custom IOs, etc.). The application IO 107A is coupledto the aggregation circuits 126 and the dispersal circuits 128. Circuitsconfigured in the programmable fabric 404 can exchange data with theapplication IO 107A through the system-level interconnect 305 and theSiP bridge 144. In some examples, the IC die 101A also includesdedicated IO 108, such as configuration IO, JTAG IO, and the like. Someor all of this dedicated IO 108 can also be coupled to the IC die 103A.

FIG. 5 is a block diagram depicting a more detailed view of the IC die104A according to an example. In particular, the programmable fabric 404is shown as including nine fabric sub-regions (FSRs) 508A, 508B, . . . ,508I (collectively FSRs 508). In general, the programmable fabric 404can include any number of FSRs. Each FSR 408 includes columns of CLBs,BRAMs, DSPs, and the like, similar to the programmable fabric shown inFIG. 3. Further, each of the FSRs 508 includes zero or more ring nodes(RNs) 504. The FSRs 508 in the center of the fabric (e.g., FSR 508E) donot include any RNs 504. The FSRs 508 along the inner edges of thefabric include one RN 504 (e.g., the FSRs 508B, 508D, 508F, and 508H).The FSRs 508 at the corners of the fabric include two RNs 504. Each RN504 includes a plurality of ring sub-nodes (RSNs) 506. As describedfurther below, the RNs 504 displace some of the fabric components in therespective FSRs 508.

The RNs 504 are coupled to wire tracks 502. In the present example, thewire tracks 502 form a ring around the programmable fabric 404. Otherinterconnect structures can be employed as discussed below. Together,the RNs 504 and the wire tracks 502 form a system-level interconnectthat can implement the system-level interconnect 350. In an example, thesystem-level interconnect supports two channel sets, one fortransmitting data from the fabric and one for receiving data to thefabric. For example, the wire tracks 502 can be configured into 256tracks, allowing for one receive channel set and one transmit channelset each having a width of 128 tracks. Of course, the wire tracks 502can be configured into more or less than 256 tracks. Each of the RNs 504is coupled to the wire tracks 502. Each RN 504 includes a router switchthat provides a bidirectional interface between the programmable fabric404 and the system-level interconnect. Circuits configured in theprogrammable fabric 404 can be coupled to specific wire tracks 502 byprogramming the RNs 504. The RNs 504 can be programmed usingconfiguration data loaded into the FPGA, or dynamically during operationof the FPGA.

Each RSN 506 comprises a repeatable portion of an RN 504. For example,if the wire tracks 502 are configured into 256 tracks, then an RN 504can include 8 RSNs 506 each controlling 32 wire tracks. Of course, an RN504 can include more or less RSNs 506 and the RSNs 506 can control moreor less than 32 wire tracks. Each RSN 506 can have the same layoutregardless of whether it is part of a RN 504 along a horizontal edge ofthe programmable fabric 404 or along a vertical edge of the programmablefabric 404. In this manner, the design tools used to design circuits forimplementation in the FPGA do not have to distinguish between RNs 504along a horizontal edge and RNs 504 along a vertical edge. An examplestructure of an RSN 506 is described below.

FIG. 6 is a block diagram depicting SiP stacks 601 and 603 according toan example. The SiP stack 601 can be part of an SiP IO (e.g., the SiP IO140 or 140A) and the SiP stack 603 can be part of another SiP IO (e.g.,the SiP IO 142 or 142A). For purposes of clarity by example, thetransport logic 114/124 is not shown.

The SiP stack 601 includes data link circuits 602 and physical circuits604. The SiP stack 603 includes data link circuits 608 implementing datalink logic and physical circuits 606 implementing physical logic. Thephysical circuits 604 and 606 form the physical layer. The data linkcircuits 602 and 608 form the data link layer.

In the example, the SiP stack 601 includes M*w source ports and M*wdestination ports. The source ports and destination ports of the datalink circuits 602 are coupled to the aggregation circuits 110 anddispersal circuits 112, respectively (e.g., the system-levelinterconnect 350). The data link circuits 602 include K*w externaloutputs and K*w external inputs. The external outputs and externalinputs of the data link circuits 602 are coupled to the physicalcircuits 604. The physical circuits 604 are coupled to the physicalcircuits 606 by the signal paths 138. Depending on the physical layer,the signal paths 138 can include approximately 2*K signal paths. Thedata link circuits 608 include K*w external inputs and K*w externaloutputs. The external inputs and outputs of the data link circuits 608are coupled to the physical circuits 606. The data link circuits 608include N*w destination ports and N*w source ports. The source ports anddestination ports of the data link circuits 608 are coupled to theaggregation circuits 126 and the dispersal circuits 128, respectively.

In operation, the data link circuits 602 multiplexes M aggregationchannels of width w into K physical channels of width w. The physicalcircuits 604 serialize the packets on the K physical channels onto thesignal paths 138. The physical circuits 606 de-serialize signals fromthe signal paths 138 into K physical channels of width w. The data linkcircuits 608 de-multiplex the K physical channels into N dispersalchannels of width w. Operation when transmitting from the data linkcircuits 608 to the data link circuits 602 (i.e., the reverse direction)is identical.

FIG. 7 is a block diagram depicting the data link layer in more detailaccording to an example. In the example of FIG. 7, M=N=3 and K=2. Thedata link circuits 602 include transmit queues 702-1 through 702-3(collectively transmit queues 702), multiplexers 704-1 and 704-2(collectively multiplexers 704), and arbitration logic 406. The datalink circuits 608 include multiplexers 706-1 through 706-3 (collectivelymultiplexers 706). Inputs of the transmit queues 702-1 through 702-3 arecoupled to source ports 708-1 through 708-3, respectively (collectivelysource ports 708). Outputs of the multiplexers 706-1 through 706-3 arecoupled to destination ports 710-1 through 710-3, respectively(collectively destination ports 710). Control inputs of the multiplexers704 are coupled to outputs of the arbitration logic 406. Inputs of thearbitration logic 406 are coupled to outputs of the transmit queues 702.In an example, the arbitration logic 406 is implemented separately fromthe SiP IO, such as in the programmable fabric 404.

In operation, the arbitration logic 406 controls the multiplexers 704 sothat there are no destination conflicts of packets transmitted acrossthe K physical channels. Each packet being transmitted includes adestination port. The arbitration logic 406 executes an arbitrationalgorithm to ensure that no two packets being transmitted in parallelacross the K physical channels have the same destination port. Examplearbitration algorithms are described below.

FIG. 8 is a flow diagram depicting a method 800 of arbitration withinthe data link layer of a SiP bridge according to an example. The method800 can be performed by the arbitration logic 406. The method 800 beginsat step 802, where the arbitration logic 406 marks all destination portsas being unused. In the example of FIG. 7, the destination portscomprise outputs 710-1 through 710-3. In general, there can be Ndestination ports, one for each input channel of the correspondingdispersal circuits.

At step 804, the arbitration logic 406 selects a multiplexer forscheduling. In the example of FIG. 7, there are two multiplexers 704-1and 704-2 to be scheduled by the arbitration logic 406. In general,there can be K multiplexers 704, one for each physical channel.

At step 806, the arbitration logic 406 identifies a destination port foreach transmit queue ready to transmit. In the example of FIG. 7, thereare three transmit queues 702-1 through 702-3. In general, there can beM transmit queues, one for each output channel of the correspondingaggregation circuit. If a transmit queue is not empty, the arbitrationlogic 406 can deem the transmit queue as ready to transmit. Each of thetransmit queues 702 stores one or more packets having a particulardestination port. The arbitration logic 406 parses the packet(s) toidentify the destination ports.

At step 808, the arbitration logic 406 schedules one or more transmitqueues targeting unused destination port(s) for transmission. In anexample, scheduling can be done in parallel using variations of amaximum matching algorithm. If a transmit queue includes packet(s) fortargeting a used destination port, the arbitration logic 406 holds thetransmit queue. At step 810, the arbitration logic 406 marks theidentified destination ports as being used. At step 812, the arbitrationlogic determines whether there are more multiplexers to schedule. If so,the method 800 returns to step 804 and selects the next multiplexer.Otherwise, the method 800 returns to step 802 and marks all destinationports as unused. As such, the arbitration logic 406 executes steps804-812 for each multiplexer to be scheduled (e.g., each of themultiplexers 704). The arbitration logic 406 executes steps 802 through812 for multiple scheduling rounds. In this manner, the arbitrationlogic 406 generates and implements schedules for controlling themultiplexers 704 so that the multiplexers 704 multiplex output of thetransmit queues 702 such that packets transmitted in parallel over thephysical channels have different destination ports and there is nocongestion or conflict at the multiplexers 706.

FIG. 9 is a flow diagram depicting a method 900 of arbitration withinthe data link layer of a SiP bridge according to another example. Themethod 900 can be performed by the arbitration logic 406. The method 900differs from the method 800 in that the arbitration logic 406 assignsweights to each of the transmit queues 702 and selects packets from thetransmit queues for transmission over the physical channels based on theweights in addition to managing destination port conflicts.

The method 900 begins at step 902, where the arbitration logic 406 marksall destination ports as being unused. In the example of FIG. 7, thedestination ports comprise outputs 710-1 through 710-3. In general,there can be N destination ports, one for each input channel of thecorresponding dispersal circuits.

At step 904, the arbitration logic 406 updates deficit counts for eachtransmit queue based on assigned weights. That is, the arbitration logic406 can assign weight to each transmit queue. Some transmit queues canhave more weight (higher priority) than other transmit queues. Thedeficit counts are used to control how many packets are selected from agiven transmit queue at a given time during the scheduling.

At step 906, the arbitration logic 406 selects a multiplexer forscheduling. In the example of FIG. 7, there are two multiplexers 704-1and 704-2 to be scheduled by the arbitration logic 406. In general,there can be K multiplexers 704, one for each physical channel.

At step 908, the arbitration logic 406 selects a transmit queue (e.g.,one of the transmit queues 702). At step 910, the arbitration logic 406determines whether the destination port targeted by packet(s) in theselected transmit queue is unused. If not, the method 900 returns tostep 908 and the arbitration logic 406 selects the next transmit queue.Otherwise, the method 900 proceeds to step 912.

At step 912, the arbitration logic 406 marks the identified destinationport as being in use. At step 914, the arbitration logic 406 schedulesthe selected transmit queue for transmission until empty or until thecorresponding deficit count satisfies a threshold. A higher deficitcount allows more packets to be selected form a given transmit queuethan a lower deficit count.

At step 916, the arbitration logic 406 determines whether the selectedtransmit queue is empty. If so, the method 900 proceeds to step 918,where the arbitration logic 406 resets the deficit count for theselected transmit queue to an initial value. If the selected transmitqueue is not empty, the method 900 proceeds instead to step 920.

At step 920, the arbitration logic 406 determines whether there are moretransmit queues to be processed in this iteration. If all the transmitqueues have been processed, the method 900 proceeds to step 922.Otherwise, the method 900 returns to step 908 and selects the nexttransmit queue. At step 922, the arbitration logic 406 determineswhether there are more multiplexers to be scheduled. If so, the method900 returns to step 906 and selects the next multiplexer. Otherwise, themethod 900 returns to step 902 and marks all destination ports asunused. As such, the arbitration logic 406 executes steps 906-922 foreach multiplexer to be scheduled (e.g., each of the multiplexers 704).The arbitration logic 406 executes steps 908-920 for each transmit queuegiven a selected multiplexer. The arbitration logic 406 executes steps902-922 for multiple scheduling rounds. In this manner, the arbitrationlogic 406 generates and implements schedules for controlling themultiplexers 704 so that the multiplexers 704 multiplex output of thetransmit queues 702 such that packets transmitted in parallel over thephysical channels have different destination ports and there is nocongestion or conflict at the multiplexers 706. The arbitration logic406 also accounts for queue priority and fairness using a weightingscheme.

FIG. 10 is a flow diagram depicting a method 1000 of transmitting datafrom a programmable IC in an IC system according to an example. Themethod 1000 may be performed by the IC die 101 to transmit data from theprogrammable fabric 404 from the IC system 400. The method 1000 beginsat step 1002, where the system-level interconnect 350 couples data tothe SiP IO 140A through its aggregation channels. In an example, theprogrammable fabric 404 includes the transport logic 114 configured todivide the data into packets (step 1004).

At step 1006, the SiP IO 140A transmits the data from the aggregationchannels over a smaller number of physical channels of the SiP bridge144. For example, at step 1008, the data link logic 116 queues data fromthe aggregation channels into transmit queues. At step 1010, the datalink logic 116 multiplexes the transmit queues among the physicalchannels, while the arbitration logic 406 manages destination conflicts.At step 1012, the arbitration logic 406 can also assign weights to thetransmit queues and select packets for transmission based on theweights.

At step 1014, the SiP IO 142 in the IC die 103 receives the data fromthe SiP bridge 144. At step 1016, the dispersal circuits 128 couple thedata from the SiP IO 142 to the application IO 107A through thedispersal channels. At step 1018, the application IO 107A consumes thedata and/or transmits the data to external circuit(s).

FIG. 11 is a flow diagram depicting a method 1100 of receiving data to aprogrammable IC in an IC system according to an example. The method 1100may be performed by the IC die 101 to receive data from the applicationcircuits 107. The method 1100 begins at step 1102, where the aggregationcircuits 126 couple data from the application circuits 107 to the SiP IO142 through aggregation channels. In an example, the SiP IO 142 includesthe transport logic 124 configured to divide the data into packets (step1104).

At step 1106, the SiP IO 142 transmits the data from the aggregationchannels over a smaller number of physical channels of the SiP bridge144. For example, at step 1108, the data link logic 122 queues data fromthe aggregation channels into transmit queues. At step 1110, the datalink logic 122 multiplexes the transmit queues among the physicalchannels, while manages destination conflicts. At step 1112, the datalink logic 122 can also assign weights to the transmit queues and selectpackets for transmission based on the weights. That is, the data linklogic 122 can include arbitration logic similar to the arbitration logic406.

At step 1114, the SiP IO 140A in the IC die 101 receives the data fromthe SiP bridge 144. At step 1116, the system-level interconnect 350couples the data from the SiP IO 140A to the programmable fabric 404through the dispersal channels. At step 1118, the programmable fabric404 consumes the data.

FIG. 12 is a block diagram depicting physical logic 1200 according to anexample implementation. The physical logic 1200 includes a plurality ofSerDes transceivers 1201, e.g., SerDes transceivers 1201-1 through1201-K. Each SerDes transceiver 1201 supports K output physical channelsand K input physical channels. Each SerDes transceiver 1201 includes aparallel interface 1204 including a plurality of signal lines, e.g.,signal lines 1204-1 through 1204-w. Each SerDes transceiver 1201includes a serial interface 1202. The parallel interfaces 1204 arecoupled to the external ports of the data link circuits, as describedabove. Each SerDes transceiver 1201 is configured to either serializeits parallel input or de-serialize its serial input.

As noted above, the IC systems described herein can use any type of USRSerDes technology that meets certain requirements. There are threeparameters of the physical layer to consider in order of priority: (1)bandwidth per pin; (2) power per bit; and (3) area. A figure of merit(FoM) can be defined for any physical logic that includes the first twofactors: FoM bandwidth per pin)/(power per bit). In an example, thephysical logic described herein can include a FoM greater than or equalto 20 (Gb/s)(pJ/bit).

FIG. 13 is a block diagram depicting an integration of the system-levelinterconnect 350 with a horizontal edge of the programmable fabric 404according to an example. In the example, the programmable fabric 404includes a fabric of interconnect elements 311 and CLEs 312. Each CLE312 includes a slice 312L and a slice 312M. The CLE slices 312L and 312Mcan include lookup tables (LUTs), multiplexers, flip-flops,combinatorial logic, and the like. The interconnect elements 111 areconfigured to couple the CLEs 312 to the programmable interconnect(shown and described above in FIG. 3).

The system-level interconnect 350 includes a horizontal edge of theprogrammable fabric 404 (horizontal fabric edge 1308) and the wiretracks 502. The horizontal fabric edge 1308 includes interconnectelements 311A and RSNs 506. Each RSN 506 occupies a region equivalent toa pair of CLEs 312L and 312M. Each RSN 506 includes switch circuitrycoupled to a portion of the wire tracks 502. In particular, each RSN 506includes clockwise (CW) links 1304 to adjacent circuitry through thewire tracks 502. Each RSN 506 also includes counter-clockwise (CCW)links 1302 to adjacent circuitry through the wire tracks 502. Theadjacent circuitry can be an RSN in another RN or the SiP IO 140Adepending on the position of the RSN 506. The interconnect elements 311Aare configured to couple the RSNs 506 to the programmable interconnectof the programmable fabric 404.

In an example, the wire tracks 502 include 256 tracks as described inthe examples above. An RSN 506 is coupled to a portion of the wiretracks 502 in both the CW and CCW directions. For example, an RSN 506can be coupled to 32 of the wire tracks 502 in each of the CW and CCWdirections (e.g., 64 total links). A plurality of the RSNs 506 combineto form an RN 504, which is coupled to all of the wire tracks 502. Forexample, if the RSNs 506 control 32 wire tracks, then a RN 504 includes8 RSNs 506.

One or more sets of RSNs 506 may be integrated into the programmablefabric 404 as shown in FIG. 13 to construct one or more RNs 504 alongthe horizontal edge. For example, each FSR 508 of the programmablefabric 404 can include one RN 504. A portion of the CLEs 312 are removedfrom the fabric edge 1308 and replaced with the circuitry of the RSNs506. A portion of the programmable interconnect (e.g., horizontal longwires) is removed to accommodate the wire tracks 502.

FIG. 14 is a block diagram depicting an integration of the system-levelinterconnect 350 with a vertical edge of the programmable fabric 404according to an example. In the example, the programmable fabric 404includes a fabric of interconnect elements 111 and CLEs 312. Each CLE312 includes a slice 312L and a slice 312M. The CLE slices 312L and 312Mcan include lookup tables (LUTs), multiplexers, flip-flops,combinatorial logic, and the like. The interconnect elements 311 areconfigured to couple the CLEs 312 to the programmable interconnect(shown and described above in FIG. 3).

The system-level interconnect 350 includes a vertical edge of theprogrammable fabric 404 (vertical fabric edge 1408) and the wire tracks502. The vertical fabric edge 1408 includes interconnect elements 311Aand RSNs 506. Each RSN 506 occupies a region equivalent to a pair ofCLEs 312L and 312M. Each RSN 506 includes switch circuitry coupled to aportion of the wire tracks 502. In particular, each RSN 506 includes CWlinks 1404 to adjacent circuitry through the wire tracks 502. Each RSN506 also includes CCW links 1402 to adjacent circuitry through the wiretracks 502. The adjacent circuitry can be an RSN in another RN or theSiP IO 140A depending on the position of the RSN 506. The interconnectelements 311A are configured to couple the RSNs 506 to the programmableinterconnect of the programmable fabric 404.

In an example, the wire tracks 502 may include 256 tracks as describedin the examples above. An RSN 506 is coupled to a portion of the wiretracks 502 in both the CW and CCW directions. For example, an RSN 506can be coupled to 32 of the wire tracks 502 in each of the CW and CCWdirections (e.g., 64 total links). A plurality of the RSNs 506 combineto form an RN 504, which is coupled to all of the wire tracks 502. Forexample, if the RSNs 506 control 32 wire tracks, then a RN 504 includes8 RSNs 506.

One or more sets of RSNs 506 may be integrated into the programmablefabric 404 as shown in FIG. 14 to construct one or more RNs 504 alongthe vertical edge. For example, each FSR 508 of the programmable fabric404 can include one RN 504 along one or both of its vertical edges. Aportion of the CLEs 312 are removed from the vertical fabric edge 1408and replaced with the circuitry of the RSNs 506. A portion of theprogrammable interconnect (e.g., vertical long wires) is removed toaccommodate the wire tracks 502.

FIG. 15 is a block diagram depicting a RSN 506 according to an example.The RSN 506 includes an arbiter 1502, buffers 1504, flip-flops 1506,flip-flops 1508, buffers 1510, buffers 1512, flip-flops 1514, amultiplexer 1516, a multiplexer 1518, and a multiplexer 1520. Thebuffers 1510 receive a 16-bit input from a left-side RSN (e.g., eitherin the same RN or an adjacent RN). Likewise, the buffers 1504 receive a16-bit input from a right-side RSN (e.g., either in the same RN or anadjacent RN) or from the SiP IO 140. Outputs of the buffers 1510 and thebuffers 1504 are coupled to inputs of the multiplexer 1518. An output ofthe multiplexer 1518 is coupled to an input of the flip-flops 1514. Theflip-flops 1514 include a 16-bit output coupled to an interconnectelement 311A.

The buffers 1512 include a 16-bit input coupled to the interconnectelement 311A. An output of the buffers 1512 is coupled to an input ofthe multiplexer 1520 and an input of the multiplexer 1516. Another inputof the multiplexer 1520 is coupled to the output of the buffers 1504.Another input of the multiplexer 1516 is coupled to the output of thebuffers 1510. An output of the multiplexer 1520 is coupled to an inputof the flip-flops 1508. An output of the multiplexer 1516 is coupled toan input of the flip-flops 1506. The flip-flops 1508 include a 16-bitoutput coupled to the left-side RSN. The flip-flops 1506 include a16-bit output coupled to the right-side RSN or the SiP IO 140A. Controlinputs of the multiplexers 1516, 1518, and 1520 are coupled to outputsof the arbiter 1502. Inputs of the arbiter 1502 are coupled to theoutputs of the buffers 1504, 1510, and 1512.

In operation, the RSN 506 buffers input from the programmable fabric 404through the interconnect element 311A using the buffers 1512. Thearbiter 1502 routes the buffered input either to the flip-flops 1508 orthe flip-flops 1506. The flip-flops 1506 are coupled to the wire tracks502 through CW links. The flip-flops 1508 are coupled to the wire tracks502 through CCW links. The flip-flops 1506 and 1508 register the datafor transmission to adjacent circuitry (e.g., either an adjacent RSN orthe SiP IO 140A). The RSN 506 also buffers input from adjacent circuitry(e.g., an adjacent RSN or the SiP IO 140A) using the buffers 1504 and1510. The buffers 1504 are coupled to the wire tracks 502 through CCWlinks. The buffers 1510 are coupled to the wire tracks 502 through CWlinks. The arbiter 1502 routes the buffered input from either thebuffers 1504 or the buffers 1510 to the flip-flops 1514. The flip-flops1514 provide registered output to the programmable fabric 404 throughthe interconnect element 311A. Thus, the RSN 506 implements a 16-bitswitch. In other examples, the RSN 504 can implement a switch having awidth less than or greater than 16 bits. The layout of the RSN 506 shownin FIG. 15 allows the same RSN implementation to be used along eitherthe horizontal edge or the vertical edge of the programmable fabric 404.

FIGS. 16A through 16F show example SIR implementations of thesystem-level interconnect 350. As shown in FIG. 16A, the wire tracks 502include a ring of tracks around the programmable fabric 404. RNs 504 areintegrated along both horizontal edges and both vertical edges of theprogrammable fabric 404. The wire tracks 502 are coupled between the RNs504 and the SiP IO 140A. Circuitry 1602 configured in the programmablefabric 404 can be coupled to the SiP IO 140A through any of the RNs 504disposed along any of the edges.

As shown in FIG. 16B, another implementation of the wire tracks (wiretracks 502A) includes a partial ring of tracks around the programmablefabric. In the example shown, the wire tracks 502A surround bothhorizontal edges and one vertical edge of the programmable fabric 404.In other examples, the wire tracks 502A can surround both vertical edgesand one horizontal edge of the programmable fabric 404. In still otherexamples, the wire tracks 502A can be adjacent to one vertical edge andone horizontal edge. The wire tracks 502A are coupled between the RNs504 and the SiP IO 140A. Circuitry 1604 configured in the programmablefabric 404 can be coupled to the SiP IO 140A through any of the RNs 504.

As shown in FIG. 16C, another implementation of the wire tracks (wiretracks 502B) includes tracks adjacent to one horizontal edge of theprogrammable fabric 404. The wire tracks 502B is coupled between the RNs504 and the SiP IO 140A. Circuitry 1606 configured in the programmablefabric 404 can be coupled to the SiP IO 140A through any of the RNs 504.

As shown in FIG. 16D, another implementation of the wire tracks (wiretracks 502C) includes tracks adjacent to one vertical edge of theprogrammable fabric 404. The wire tracks 502C are coupled between theRNs 504 and the SiP IO 140A. Circuitry 1608 configured in theprogrammable fabric 404 can be coupled to the SiP IO 140A through any ofthe RNs 504.

As shown in FIG. 16E, another implementation of an SIR includes two setsof wire tracks 502A and 502C. The wire tracks 502A are adjacent to oneor more sides of the programmable fabric 404 (e.g., two horizontal andone vertical side as shown). The wire tracks 502C are adjacent to one ormore sides of the programmable fabric 404 (e.g., one vertical side asshown). The wire tracks 502A are coupled between RNs 504 and the SiP IO140A. The wire tracks 502C are coupled between another set of RNs 504and another SiP IO circuit 104B. Circuitry 1610 configured in theprogrammable fabric 404 can be coupled to the SiP IO 140A through any ofthe RNs 504 coupled to the wire tracks 502A. Circuitry 1612 configuredin the programmable fabric 404 can be coupled to the SiP IO circuit 140Bthrough any of the RNs 504 coupled to the wire tracks 502C.

As shown in FIG. 16F, another implementation of an SIR includes a meshof wire tracks 502 (e.g., wire tracks 502-1 through 502-4). Each set ofwire tracks 502 surrounds one or more FSRs 508 of the programmablefabric 404. The wire tracks 502 are interconnected as a mesh. Each FSR508 includes RNs 504 disposed along one or more edges thereof. The RNs504 are coupled to the respective wire tracks 502-1 through 502-4. Themesh of wire tracks 502 are coupled to the SiP IO 140A. Circuitry (notshown) configured in any of the FSRs 508 can be coupled to the SiP IO140A through one or more RNs 504 and one or more wire tracks 502-1through 502-4. While four wire track rings are shown, the SIR caninclude more or less than four wire track rings.

FIG. 17 is a flow diagram depicting a method 1700 of transmitting datafrom a programmable IC in an IC system according to an example. Themethod 1700 may be performed by the IC die 101 to transmit data from theprogrammable fabric 404 from the IC system 400. The method 1700 beginsat step 1702, circuitry in the programmable fabric couples data to ringnode(s) 504 integrated with edges of the programmable fabric. At step1703, the ring node(s) 504 couple the data to the SiP IO 140A throughchannels of the wire tracks 502. In an example, the programmable fabric404 includes the transport logic 114 configured to divide the data intopackets (step 1704).

At step 1706, the SiP IO 140A transmits the data from the bus channelsover a smaller number of physical channels of the SiP bridge 144. Forexample, at step 1708, the data link logic 116 queues data from the buschannels into transmit queues. At step 1710, the data link logic 116multiplexes the transmit queues among the physical channels, while thearbitration logic 406 manages destination conflicts. At step 1712, thearbitration logic 406 can also assign weights to the transmit queues andselect packets for transmission based on the weights.

At step 1714, the SiP IO 142 in the IC die 103 receives the data fromthe SiP bridge 144. At step 1716, the dispersal circuits 128 couple thedata from the SiP IO 142 to the application IO 107A through thedispersal channels. At step 1718, the application IO 107A transmits thedata to external circuit(s).

While the foregoing is directed to specific examples, other and furtherexamples may be devised without departing from the basic scope thereof,and the scope thereof is determined by the claims that follow.

What is claimed is:
 1. An integrated circuit (IC) system, comprising: apackage substrate having a programmable integrated circuit (IC) die anda companion IC die mounted thereon, the programmable IC die including aprogrammable fabric and the companion IC die including applicationcircuitry; a system-in-package (SiP) bridge including a first SiP IOcircuit disposed in the programmable IC die, a second SiP IO circuitdisposed in the companion IC die, and conductive interconnect on thepackage substrate electrically coupling the first SiP IO circuit and thesecond SiP IO circuit; first aggregation and first dispersal circuits inthe programmable IC die coupled between the programmable fabric and thefirst SiP IO circuit; and second aggregation and second dispersalcircuits in the companion IC die coupled between the application IOcircuitry and the second SiP IO circuit; wherein the first and secondSiP IO circuits are configured to multiplex multi-channel output of thefirst and second aggregation circuits, respectively, onto a firstplurality of physical channels implemented over the conductiveinterconnect; and de-multiplex input from a second plurality of physicalchannels implemented over the conductive interconnect onto multi-channelinput of the first and second dispersal circuits, respectively.
 2. TheIC system of claim 1, wherein the first aggregation and the firstdispersal circuits comprise a system level system-level interconnectcoupled between a programmable interconnect of the programmable fabricand the first SiP IO circuit.
 3. The IC system of claim 2, wherein thesystem-level interconnect comprises a network-on-chip (NoC).
 4. The ICsystem of claim 1, wherein the programmable IC die comprises a directconnection to the companion IC die separate from the SiP bridge.
 5. TheIC system of claim 1, wherein the programmable IC die includesarbitration logic, and wherein the first SiP IO circuit comprises a datalink circuit and a transceiver circuit, where: an internal interface ofthe data link circuit is coupled to the first aggregation and the firstdispersal circuits; an external interface of the data link circuit iscoupled to an internal interface of the transceiver circuit; an externalinterface of the transceiver circuit is coupled to the conductiveinterconnect; and a control interface of the data link circuit iscoupled to the arbitration logic.
 6. The IC system of claim 5, whereinthe arbitration logic is implemented within the programmable fabric ofthe programmable IC die.
 7. The IC system of claim 1, wherein theprogrammable IC die includes transport logic configured to packetizedata transmitted to the first aggregation circuit and de-packetize datareceived from the first dispersal circuit.
 8. The IC system of claim 7,wherein the transport logic is implemented within the programmablefabric of the programmable IC die.
 9. A programmable integrated circuit(IC), comprising: a system-in-package (SiP) input/output (IO) circuitconfigured to be coupled to a companion IC through external conductiveinterconnect; a programmable fabric without at least a portion ofapplication circuitry; and aggregation and dispersal circuits coupledbetween the programmable fabric and the SiP IO circuit; wherein theaggregation and the dispersal circuits comprise a system-levelinterconnect coupled between a programmable interconnect of theprogrammable fabric and the SiP IO circuit; and wherein the system-levelinterconnect comprises a network-on-chip (NoC).
 10. The programmable ICof claim 9, wherein the programmable fabric is directly connected to thecompanion IC separate from the SiP IO circuit.
 11. The programmable ICof claim 9, wherein the programmable fabric is configured to implementarbitration logic, and wherein the SiP IO circuit comprises a data linkcircuit and a transceiver circuit, where: an internal interface of thedata link circuit is coupled to the aggregation and the dispersalcircuits; an external interface of the data link circuit is coupled toan internal interface of the transceiver circuit; an external interfaceof the transceiver circuit is coupled to the external conductiveinterconnect; and a control interface of the data link circuit iscoupled to the arbitration logic.
 12. The programmable IC of claim 9,wherein the programmable fabric is configured to implement transportlogic that packetizes data transmitted to the aggregation circuit andde-packetizes data received from the dispersal circuit.
 13. A method oftransmitting data from a programmable integrated circuit (IC) in an ICsystem, the method comprising: coupling the data to a firstsystem-in-package (SiP) IO circuit through a plurality of channels of anaggregation circuit in the programmable IC; transmitting the data fromthe plurality of channels by multiplexing the data over a smaller numberof physical channels implemented over a conductive interconnect betweenthe programmable IC and a companion IC; receiving the data from theplurality of physical channels at a second SiP IO circuit in thecompanion IC; and coupling the data from the second SiP IO circuit toapplication circuitry in the companion IC by demultiplexing the datathrough a plurality of channels of a dispersal circuit in the companionIC.
 14. The method of claim 13, wherein the data is divided intopackets.
 15. The method of claim 14, wherein the second SiP IO circuitincludes a plurality of internal output ports coupled to the respectiveplurality of channels of the dispersal circuit, and wherein the packetseach have a destination port selected from one of the plurality ofinternal output ports.
 16. The method of claim 15, wherein the step oftransmitting comprises: queuing the data from the plurality of channelsof the aggregation circuit in a respective plurality of transmit queues;and multiplexing output of the respective plurality of transmit queuesamong the plurality of physical channels such that packets transmittedin parallel over the plurality of physical channels have differentdestination ports.
 17. The method of claim 16, wherein the step ofmultiplexing further comprises: assigning weights to each of therespective plurality of transmit queues; and selecting packets from therespective plurality of transmit queues for transmission over theplurality of physical channels based on the weights.
 18. An integratedcircuit (IC) system, comprising: a package substrate having aprogrammable integrated circuit (IC) die and a companion IC die mountedthereon, the programmable IC die including a programmable fabric and thecompanion IC die including application circuitry; a system-in-package(SiP) bridge including a first SiP IO circuit disposed in theprogrammable IC die, a second SiP IO circuit disposed in the companionIC die, and conductive interconnect on the package substrateelectrically coupling the first SiP IO circuit and the second SiP IOcircuit; first aggregation and first dispersal circuits in theprogrammable IC die coupled between the programmable fabric and thefirst SiP IO circuit; and second aggregation and second dispersalcircuits in the companion IC die coupled between the application IOcircuitry and the second SiP IO circuit; wherein the programmable IC diecomprises a direct connection to the companion IC die separate from theSiP bridge.
 19. An integrated circuit (IC) system, comprising: a packagesubstrate having a programmable integrated circuit (IC) die and acompanion IC die mounted thereon, the programmable IC die including aprogrammable fabric and the companion IC die including applicationcircuitry; a system-in-package (SiP) bridge including a first SiP IOcircuit disposed in the programmable IC die, a second SiP IO circuitdisposed in the companion IC die, and conductive interconnect on thepackage substrate electrically coupling the first SiP IO circuit and thesecond SiP IO circuit; first aggregation and first dispersal circuits inthe programmable IC die coupled between the programmable fabric and thefirst SiP IO circuit; and second aggregation and second dispersalcircuits in the companion IC die coupled between the application IOcircuitry and the second SiP IO circuit; wherein the programmable IC dieincludes arbitration logic, and wherein the first SiP IO circuitcomprises a data link circuit and a transceiver circuit, where: aninternal interface of the data link circuit is coupled to the firstaggregation and the first dispersal circuits; an external interface ofthe data link circuit is coupled to an internal interface of thetransceiver circuit; an external interface of the transceiver circuit iscoupled to the conductive interconnect; and a control interface of thedata link circuit is coupled to the arbitration logic.
 20. An integratedcircuit (IC) system, comprising: a package substrate having aprogrammable integrated circuit (IC) die and a companion IC die mountedthereon, the programmable IC die including a programmable fabric and thecompanion IC die including application input/output (IO) circuitry,wherein the programmable IC die does not include application IOcircuitry; a system-in-package (SiP) bridge including a first SiP IOcircuit disposed in the programmable IC die, a second SiP IO circuitdisposed in the companion IC die, and conductive interconnect on thepackage substrate electrically coupling the first SiP IO circuit and thesecond SiP IO circuit; first aggregation and first dispersal circuits inthe programmable IC die coupled between the programmable fabric and thefirst SiP IO circuit; and second aggregation and second dispersalcircuits in the companion IC die coupled between the application IOcircuitry and the second SiP IO circuit; wherein the first and secondSiP IO circuits are configured to multiplex multi-channel output of thefirst and second aggregation circuits, respectively, onto a plurality ofphysical channels implemented over the conductive interconnect; andde-multiplex input from the plurality of physical channels implementedover the conductive interconnect onto multi-channel input of the firstand second dispersal circuits, respectively.
 21. The IC system of claim20, wherein circuits in the programmable fabric are configured to usethe first and second SiP IO circuits to communicate with the applicationIO circuitry in the companion IC die.
 22. The IC system of claim 21,wherein the programmable IC die further comprises dedicated IO circuitryconfigured for at least one of programming or testing the programmableIC die, wherein the circuits in the programmable fabric that areconfigured to use the application IO circuitry in the companion IC diedo not use the dedicated IO circuitry.